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Summary 

No  other  time  in  history  have  we  seen  as  much  readily  available  technology  for  training  warfighters  as  we 
see  today.  There  is  an  abundance  of  training  technology  in  almost  every  warfighter  domain.  However,  the 
challenge  still  remains  as  to  how  best  to  use  this  technology,  and  in  particular,  what  tasks  can  benefit  the 
most  from  insertion  of  technology  into  the  training  environment.  The  Air  Force  Research  Laboratory 
(AFRL)  in  Mesa,  Arizona  recently  made  substantial  progress  toward  this  challenge  by  identifying  how 
simulation  technologies  could  be  used  to  enhance  flying  within  the  U.S.  Air  Force,  Air  Education  and 
Training  Command  (AETC).  Researchers  from  AFRL  developed  a  technology  insertion  process  that 
included  instructor  pilot  workshops,  visits  to  flying  training  bases,  training  task  definitions,  assessment  of 
current  simulators,  surveys  to  assess  critical  training  tasks,  and  a  Quality  Function  Deployment  (QFD)  to 
prioritize  training  technologies.  The  workshops  were  a  critical  component  of  the  technology  insertion 
process  as  they  helped  us  identify  the  training  requirements  and  tasks  for  several  aircraft  training  courses 
including  the  T-6,  T-38,  T-1A,  AT-38B,  T-38C,  F-15,  and  F-16.  After  the  instructor  pilots  validated  the 
task  requirements,  AFRL  launched  an  internet-based  survey  to  over  700  instructor  pilots  across  the  U.S. 
Air  Force.  The  survey  was  designed  to  collect  ratings  on  task  difficulty,  syllabus  time  allocation,  how 
often  tasks  contribute  to  busted  check  rides,  proficiency  of  graduates,  and  the  adequacy  of  current 
simulation  devices.  This  data  identified  the  more  critical  training  needs  and  formed  the  basis  of  an  index 
for  weighting  the  criteria  in  the  subsequent  QFD.  QFD  workshop  participants  included  former  and 
current  instructor  pilots,  researchers,  and  engineers  who  have  had  exposure  to  updated  training 
technologies.  Participants  in  the  QFD  workshop  assessed  how  useful  27  different  advanced  simulation 
technologies  would  be  for  training  each  task.  The  priority  weights  from  the  survey  were  then  used  as 
multipliers  for  each  technology  score  to  yield  total  weighted  scores  for  each  technology  and  a  prioritized 
rank  ordering  of  those  technologies.  This  analytic  process  yielded  valuable  information  to  aid  leaders  in 
making  decisions  about  technology  investment.  In  this  paper,  we  focus  on  the  technology  insertion 
process  and  how  this  process  can  be  used  to  prioritize  advanced  simulation  technology  in  a  flying  training 
environment. 

1.0  Backround 

In  the  last  few  years,  rapid  advances  in  simulation  technology  have  provided  more  capability,  especially 
in  the  areas  of  visual  systems,  networking,  and  realistic  databases.  However,  a  great  deal  of  the  current 
simulation  training  technology  in  the  United  States  Air  Force  (USAF)  has  not  been  updated  to  take 
advantage  of  efficiencies  offered  by  some  of  these  newer  technologies. 

Paper  presented  at  the  RTO  HFM  Symposium  on  “Advanced  Technologies  for  Military  Training  ”, 
held  Genoa,  Italyl3  -  15  October  2003,  and  published  in  RTO-MP-HFM-101. 
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In  November  2001,  the  USAF  Air  Education  and  Training  Command  (AETC)  asked  the  Air  Force 
Research  Laboratory  (AFRL)  to  assess  the  current  state  of  flying  training  in  AETC  and  to  identify  and 
analyze  opportunities  where  advanced  simulation  technologies  could  be  used  to  increase  the  efficiency 
and  effectiveness  of  AETC  flying  training  programs.  The  Command’s  motivation  for  the  study  stemmed 
from  a  desire  to  build  a  simulation  technology  integration  roadmap  that  will  guide  the  development  and 
enhancement  of  flying  training.  The  results  of  the  study  will  assist  modernization  planners  in  the 
development  of  a  technology  integration  roadmap  that  will  identify  the  plan  for  insertion  of  those 
technologies  with  the  greatest  potential  to  assist  in  the  capability  to  train,  rehearse,  refine,  and  remediate 
tasks  in  a  safe,  effective  instructional  simulation  environment. 

Systematically  gathering  data  on  training  needs  allowed  for  the  prioritization  of  advanced  training 
technologies  to  be  implemented  across  AETC.  This  effort  provided  a  first  look  at  potential  targets  of 
modem  and  high  fidelity  training  technology  insertion  in  AETC  Specialized  Undergraduate  Pilot  Training 
(SUPT),  Introduction  to  Fighter  Fundamentals  (IFF),  and  fighter  Flying  Training  Units  (FTUs).  These 
training  system  enhancements  could  help  AETC  by: 

•  Reducing  the  number  of  busted  check-rides,  student  wash-backs,  and  student  wash-outs 

•  Training  students  to  a  higher  level  of  proficiency  than  current  course  outputs 

•  Ultimately  producing  pilots  that  are  more  combat  mission  ready  due  to  a  higher  level  of  mastery 
across  a  greater  number  of  skills,  tasks,  and  competencies 

This  study  provided  quantitative  and  qualitative  survey  data  from  the  targeted  IP  populations,  onsite 
evaluations  of  all  ground-based  simulator  systems  flown  in  AETC,  and  technology  utility  assessment 
scores  from  current  instructor  pilots.  Scientists,  engineers,  and  subject-matter  experts  from  AFRL  have 
made  recommendations  on  technology  applications  based  on  analysis  of  the  data  and  knowledge  of  the 
current  state  of  the  art  for  the  various  technologies  being  considered. 

2.0  Evaluations  of  Current  Simulators 

One  of  the  critical  aspects  of  this  study  was  an  evaluation  of  the  current  simulation  technologies  being 
used  at  operational  training  sites.  Scientists  and  engineers  from  AFRL  visited  six  different  training  bases 
that  have  representative  simulators  for  specialized  undergraduate  pilot  training  (SUPT),  Euro-NATO  Joint 
Jet  Pilot  Training  (ENJJPT),  and  fighter  aircraft  flying  training  units  (FTUs).  The  six  bases  visited 
allowed  the  research  team  to  assess  all  the  different  training  devices  being  used  for  the  aircraft  included  in 
the  study. 

The  objective  of  the  simulator  site  visits  was  to  evaluate  the  current  state-of-the  art  for  the  simulators  in 
AETC.  The  evaluation  team  constructed  two  evaluation  tools  to  use  in  the  evaluation.  The  first  was  a 
mission  profile  list  called  AETC  Simulator  Evaluation  Procedures.  This  list  contained  the  major  mission 
areas  the  simulator  operator  would  need  to  present  during  the  evaluation  mission  such  as  takeoff,  landing, 
night  operations,  and  formation  flying.  This  mission  profile  list  was  given  to  the  technician  or  instructor 
operating  the  console.  The  second  document  was  a  simulator  characteristic  and  technical  performance 
checklist  called  the  Simulator  Technology  Assessment  Questionnaire.  When  filled  out,  it  contained  the 
physical  description  of  the  simulator  system  and  the  performance  data  observed,  measured,  or  gathered  by 
interview,  during  the  pre-flight,  mission  flight,  and  post-mission  debrief. 

While  a  sizeable  segment  of  the  fleet  of  simulation  devices  in  AETC  is  old  by  many  standards,  the 
architecture  on  which  most  of  them  are  based  (  VME  or  PC)  makes  them  potential  candidates  for  upgrades 
that  could  be  fairly  easily  integrated  into  existing  training  systems,  thereby  significantly  reducing  the  cost 
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to  rebuild  the  devices.  The  technology  throughout  the  fleet  ranged  from  high  fidelity  cockpits  with  no 
visuals  of  any  kind  to  trainers  with  full  field-of-regard  visual  systems. 

2.1  On-site  Evaluation  Procedures 

Each  simulator  was  evaluated  using  the  same  set  of  mission  profiles  and  the  performance  evaluation 
checklist.  The  mission  profile  checklist  given  to  the  console  operator  is  shown  in  Table  1.  A  brief 
explanation  of  the  targeted  evaluation  is  included. 

Table  1.  AETC  Simulator  Evaluation  Procedures 


AETC  Simulator  Evaluation  Procedures 

Procedure 

Description 

1.  Day  VFR  takeoff 

During  this  phase,  the  ground  environment  was  evaluated  for  scene  fidelity, 
brightness,  density,  realism,  and  cultural  features. 

2.  Pitch-out  and  landing  and 
touch  and  go 

Immediately  following  takeoff,  the  simulator  was  maneuvered  for  line-up  on 
initial  at  pattern  altitude  and  a  pitchout  to  a  touch-and-go  was  accomplished 
if  the  system  had  a  visual.  In  those  cases  where  the  visual  was  of  limited 
field-of-view,  the  downwind  was  flown  to  a  radial  and  DME  to  begin  the 
base  to  final  turn  in  order  to  see  if  it  was  remotely  possible,  in  a  student 
environment,  to  use  the  simulator  for  transition  training.  This  pass  was  also 
used  to  evaluate  the  richness  of  the  database  at  the  home  field. 

3.  Climb  to  altitude  and 
descend  to  low  altitude  for 
texture  and  scene  evaluation 

Following  the  touch  and  go,  the  simulator  was  zoomed  to  altitudes  above 
20,000  feet  MSL.  At  altitude,  the  database  was  assessed  for  range  ring 
settings,  distance  to  the  horizon,  texture  pattern  interference,  anti-aliasing, 
scene  “popping”,  and  level  of  fidelity,  in  general.  Following  this  initial 
look,  the  aircraft  was  dived  towards  the  ground  to  a  very  low  level  to  check 
the  performance  of  the  image  generator  and  the  effects,  if  any,  of  resolution 
changes  in  the  out-the-window  scene  during  the  descent. 

4.  Low  altitude  flying  for 
system  performance 
evaluation 

At  very  low  altitudes,  the  simulator  was  placed  in  maximum  power  and 
flown  across  the  database  to  assess  the  performance  of  the  visual  system 
with  regards  to  frame  rate  effects,  loss  of  frame  rate,  ability  to  judge  height, 
and  the  effects  of  high  angular  rate  aileron  rolls  over  the  terrain,  in  general, 
and  over  the  home  station  airfield  with  cultural  features  (3D)  in  the  scene. 

5.  While  low  altitude  test 
collision  detection  with 
terrain  and  features 

Following  the  low  altitude  assessments  in  step  4  above,  the  simulator  was 
descended  into  the  ground  to  determine  if  collision  detection  was  present. 

In  most  implementations  when  a  collision  with  the  ground  occurs,  the 
simulator  should  halt  permanently  or  in  temporary  freeze  state  releasable  by 
the  IP.  In  higher  fidelity  simulations,  these  same  characteristics  should  be 
present  when  colliding  with  cultural  (3D)  models  attached  to  the  terrain 
surface  of  the  database.  It  should  be  noted  that  collision  detection  is  also  a 
feature  that  can  be  selected  or  de-selected,  depending  on  the  training 
objectives  of  the  mission. 

6.  Rejoin  with  other  target(s) 
if  capable 

If  the  simulator  was  capable  of  generating  another  target  in  flight,  it  was 
then  inserted  on  the  fly.  If  the  system  was  not  capable  of  doing  this,  the 
system  was  reinitialized  with  the  second  aircraft  in  the  scenario.  The  target 
was  then  located  visually  or  using  available  sensors  and  rejoined  using  the 
standard  turning  rejoin  procedures  or  straight-ahead  rejoins  if  the  target 
could  not  easily  be  turned.  The  purpose  of  this  phase  was  to  evaluate  the 
visual  image  for  resolution,  the  effects  of  frame  rate,  the  fidelity  of  the 
aircraft  models,  and  the  ability  of  the  host  and  image  generator  systems  to 
produce  a  stable,  non-jittering  target  when  close  aboard. 

7.  Evaluate  occulting  visually 

This  task  was  intended  to  assess  the  whether  or  not  another  aircraft  would 
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and  in  avionics,  if  applicable 

disappear  from  sight  visually  when  flying  behind  another  cultural  feature 
such  as  a  building  or  behind  a  terrain  feature  such  as  a  mountain  or  ridge 
line.  In  addition,  if  the  simulator  being  evaluated  had  radar,  the  same  test 
was  to  be  accomplished  in  order  to  determine  if  the  other  aircraft  would 
“disappear”  electronically  from  the  sensor  being  used  to  detect  it.  Due  to 
the  wide  range  of  fidelity  throughout  the  “fleet”,  the  lack  of  the  ability  to 
easily  maneuver  computer  generated  entities  in  most  of  the  AETC 
simulators,  and  the  lack  of  networked  targets  to  control  in  real-time,  this 
area  was  assessed  mainly  by  interviewing  onsite  or  technical  support  center 
personnel. 

8.  Fly  close  formation  with 
another  aircraft,  if  able 

Since  most  current  systems  have  a  reasonably  high  transport  delay  (time 
from  stick  input  to  measurable  state  change)  and  the  flight  aerodynamic 
model  may  not  be  exactly  like  the  aircraft,  this  phase  was  not  used  to 
formally  evaluate  the  ability  of  the  simulator  system  to  fly  close  formation. 
However,  once  aboard,  the  evaluator  did  fly  in  formation  for  a  short  period 
of  time  to  assess  other  performance  characteristics  and  to  judge  whether  or 
not  the  system  had  the  potential  to  operate  in  either  a  close,  route,  or 
extended  trail  formation  environment. 

9.  Night  approach 

Upon  completion  of  the  day  VFR  tasks  in  Steps  1-8  above,  the  simulation 
was  reconfigured  on  the  fly  or  reinitialized  to  a  night  time  environment.  The 
first  step  was  to  evaluate  the  database  fidelity,  in  general.  That  is,  were 
there  any  visual  clues  to  identify  the  ground  environment,  first,  away  from 
home  field  and,  second,  at  the  home  field  or  primary  night  approach 
airfield?  The  evaluation  also  looked  at  the  night  sky  to  determine  if  any 
special  features  such  as  clouds  and  stars  were  present  or  available.  Prior  to 
leaving  the  night  environment,  a  visual  and  an  instrument  approach  were 
flown  to  the  home  field  or  practice  instrument  approach  airfield.  If  the 
visual  approach  went  well  (no  evaluator  error),  the  instrument  approach  was 
accomplished  with  weather  effects.  The  fidelity  of  the  airfield  environment 
in  a  night  scene  was  also  noted. 

10.  Experience  special  effects 

If  the  simulation  was  capable,  any  special  effects  such  as  airburst,  sun  glint, 
lightning  flashes,  and  missile  smoke  trails  were  implemented  and  evaluated. 
Most  of  the  systems  evaluated  either  had  limited  special  effects  available  or 
none  at  all. 

11.  Air  refuel,  if  capable 

While  no  system  in  AETC  had  a  fully  operational  air-to-air  refueling 
simulation  to  include  an  articulated  tanker  boom,  this  portion  of  the 
evaluation  consisted  of  rejoining  on  a  tanker  model,  if  available,  or  another 
aircraft  such  as  the  T-l,  to  observe  the  degree  to  which  this  phase  of  training 
could  be  simulated.  Tanker  (or  other  aircraft)  fidelity  and  stability  were  two 
of  the  key  parameters  assessed. 

12.  Weather  effects,  if  capable 

During  some  point  in  the  evaluation  mission,  a  wide  range  of  weather 
effects,  as  available,  were  instantiated  and  evaluated.  This  included  general 
scene  content  when  simply  flying  VFR,  the  ability  to  invoke  cloud  layers, 
the  physical  nature  of  the  cloud  layers  (did  they  have  thickness  for 
example),  the  appearance  of  the  weather  effect,  and  the  degree  of  simulation 
of  fog  and  low  visibility  situations.  To  the  best  of  our  knowledge,  no 
simulation  system  evaluated  had  the  ability  to  present  a  physics-based 
environment  where  the  weather  phenomena  simulated  anything  beyond  just 
the  visual  effects.  It  should  be  noted  at  this  point,  however,  that  we  know  of 
NO  real-time  simulation  used  for  training  or  research  that  does  present  a 
totally  physics-based  representation  of  the  synthetic  natural  environment. 

13.  Full  stop  landing 

After  completion  of  the  steps  above,  to  the  degree  possible  based  on  time 
available  and  system  capability,  the  mission  was  terminated  with  a  day  VFR 
straight-in,  full-stop  landing.  One  last  look  was  given  to  the  airfield 
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environment,  the  aerodynamic  response  of  the  host  simulation  system  in  the 
landing  configuration,  and  the  ability  to  land  in  the  proper  touch  down  zone 
and  aero  brake,  if  appropriate. 

14.  Discuss  Instructor 

Operator  Station 

During  mission  execution,  one  of  the  AFRL  evaluators  was  positioned  at  the 
instructor  operator  station  (IOS)  to  observe  and  discuss  the  capabilities  and 
ease  of  use  of  the  IOS.  Any  further  information  needed  by  the  flight 
evaluator  was  gathered  by  interview  and  demonstration  following  the 
simulator  flight. 

15.  Discussions  with 
contractors  and  users 

Before  leaving  a  site,  an  attempt  was  made  to  gather  any  technical  data  not 
easily  available  from  discussion  and  gather  any  useful  information  pertinent 
to  the  technical  characteristics  or  performance  of  those  systems  being 
evaluated.  Where  IPs  (contractor  or  USAF)  and/or  students  were  available, 
the  system  just  flown  was  discussed  with  them. 

The  Simulator  Technology  Assessment  Questionnaire  was  used  to  gather  the  physical  and  performance 
characteristics  of  each  AETC  simulation  system.  The  information  collection  was  grouped  into  seven 
separate  categories:  System  Identification,  Visual  Display  System,  Image  Generator/Databases,  Typical 
Problem  Areas  observed  During  Flight,  Adequacy/Capability  for  Training  Characteristics,  Cockpit,  and 
Instructor  Operator  Station.  The  physical  characteristics  were  usually  a  fill-in-the -blank  answer  and 
included  items  like  image -generator  frame  rates  and  display  brightness.  The  performance  data  collected 
during  an  evaluation  flight  were  answered  with  a  simple  “yes”  or  “no”  assessment  and  looked  for  the 
existence  of  performance  characteristics  like  double  imaging  or  anti-aliasing. 

The  information  collected  on  current  training  system  architectures,  features,  capabilities,  and  performance 
characteristics  helped  the  researchers  determine  an  appropriate  list  of  candidate  technologies  to  be 
considered  and  evaluated  by  IPs  later  in  the  study. 

3.0  Instructor  Pilot  Task  Survey 

The  entire  analysis  of  the  study  hinged  on  developing  a  comprehensive  understanding  of  the  current  state 
of  training  in  AETC  and  identification  of  the  greatest  training  challenges  or  shortfalls.  To  gain  this 
insight,  the  researchers  constructed  task  surveys  for  each  of  the  flying  training  courses  in  order  to  assess 
the  current  training  programs  and  how  well  the  existing  training  device  technologies  were  meeting  their 
needs. 

Current  instructor  pilot  (IP)  involvement  was  essential  for  creating  task  surveys  that  were  valid  and 
understandable  to  each  of  the  instructor  populations.  To  help  the  researchers  identify  the  appropriate 
flying  tasks  to  use  on  the  surveys  as  well  as  develop  appropriate  survey  items,  the  researchers  conducted  a 
three-day  workshop  with  current  IPs  from  each  of  the  different  aircraft  training  courses  being  analyzed  in 
this  study.  Table  2  summarizes  the  instructor  qualifications  and  experience  levels. 

Table  2.  IP  Experience  at  Survey  Development  Workshop 


Aircraft 

Training  Course 

Base 

Primary 

Aircraft 

Military  Flight 
Hours 

T-6 

SUPT 

Moody  AFB 

B-52 

4150 

T-l 

SUPT 

Vance  AFB 

KC-135 

2500 

SUPT 

Laughlin  AFB 

C-5 

2800 

T-38A 

SUPT 

Columbus  AFB 

FAIP 

600 

SUPT 

Vance  AFB 

F-16 

1100 

ENJJPT 

Sheppard  AFB 

F-15 

2500 
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AT-38B 

IFF 

Sheppard  AFB 

A-10 

4216 

T-38C 

IFF  (Pilot  and  WSO) 

Moody  AFB 

F-15E 

1200 

F-15 

B -Course 

Tyndall 

F-15 

1100 

B -Course 

Mesa  Research  Site 
(Retired  F-15  pilot  /  IP) 

F-15 

3400 

F-16 

B -Course 

Luke  AFB 

F-16 

2300 

B -Course 

Luke  AFB 

F-16 

3400 

FAC-A,  MANTIRN, 
NVG 

Luke  AFB 

F-16 

2600 

Average  Hours 

2451 

After  the  survey  development  workshop,  AFRL  used  its  Internet-based  survey  tools  to  host  the  surveys  on 
the  Internet.  The  workshop  IPs  then  had  a  chance  to  review  and  test  the  on-line  version.  To  further 
ensure  survey  clarity,  usability,  and  thoroughness  the  researchers  traveled  to  various  training  bases  to 
field-test  the  survey  with  small  samples  (at  least  two  IPs)  from  each  of  the  targeted  survey  populations. 
As  a  result  of  the  field-testing,  the  wording  of  three  of  the  questions  and  scales  were  clarified  and  the  task 
lists  for  three  aircraft  were  slightly  modified. 

Table  3  summarizes  the  final  version  of  the  five  questions  and  rating  scales  that  were  used  on  the  surveys 
for  all  of  the  different  aircraft  training  courses  included  in  this  study. 


Table  3.  Task  Survey  Rating  Scales 


Question 

Scale 

1.  How  difficult  is  it  for  students  to  learn  this  task? 

1  =  Not  difficult 

2 

3  =  Difficult 

4 

5  =  Extremely  Difficult 

2.  How  adequate  is  the  amount  of  time  currently  allocated  in  the 
syllabus  for  the  training  of  this  task? 

1  =  Not  enough 

2 

3  =  Adequate 

4 

5  =  Too  much 

3.  How  frequently  does  this  task  contribute  to  busted  rides? 

1  =  Rarely 

2 

3  =  Occasionally 

4 

5  =  Frequently 

4.  How  well  trained  are  typical  graduates  of  your  course  on  this  task? 

1  =  Barely  proficient 

2 

3  =  Proficient 

4 

5  =  Mastered 

5.  How  well  do  current  training  simulation  devices  prepare  students 
for  this  task? 

0  =  No  device  available/applicable 

1  =  Poorly 

2 

3  =  Adequately 

4 

5  =  Extremely  well 
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These  survey  questions  were  designed  to  collect  information  to  help  determine  how  well  training  is 
currently  being  conducted  and  identify  those  areas  in  which  training  is  more  difficult  or  problematic.  The 
data  collected  from  these  surveys  formed  the  basis  for  a  priority  weighting  scheme.  A  priority  weighting 
factor  was  created  for  each  task  that  gave  stronger  weights  to  those  tasks  shown  to  be  more  difficult  or 
problematic.  Tasks  with  a  higher  priority  weighting  factor  are  greater  targets  of  opportunity  for 
enhancement  by  improved  training  simulation  technologies.  In  the  section  on  Quality  Function 
Deployment,  the  use  of  the  priority  weighting  factor  will  be  further  explained. 

In  addition  to  forming  the  basis  for  the  technology  investment  prioritization,  the  survey  collected  a  wealth 
of  data  that  is  expected  to  also  be  useful  for  those  instructors  and  commanders  making  syllabus  revisions 
or  for  helping  decision  makers  understand  the  proficiency  outputs  of  students  in  the  flying  training 
courses. 

3.1  Data  Collection 

After  field-testing  of  the  surveys,  they  were  administered  on-line  via  the  Internet  to  current  IPs  at  each  of 
the  AETC  SUPT,  ENJJPT,  IFF,  and  FTU  bases.  Table  4  lists  the  bases  and  training  courses  included  in 
the  study. 

Table  4.  Targeted  Survey  Populations 


Base 

Aircraft 

Training  Course 

Laughlin  AFB  TX 

T-38,  T-l 

SUPT 

Vance  AFB  OK 

T-38,  T-l 

JSUPT 

Columbus  AFB  MS 

T-38,  T-l 

SUPT 

Sheppard  AFB  TX 

T-38,  AT-38B 

ENJJPT,  IFF 

Moody  AFB  GA 

T-6,  T-38C 

SUPT,  IFF 

Luke  AFB  AZ 

F-16 

B-course,  NVG,  FAC -A,  and  MANTIRN 

Tucson  ANG  AZ 

F-16 

B-course,  NVG 

Springfield  ANG  OEI 

F-16 

B-course,  NVG 

Kelly  ANG  TX 

F-16 

B-course,  NVG 

Tyndall  AFB  FL 

F-15 

B-course 

Kingsley  ANG  OR 

F-15 

B-course,  NVG 

Official  tasking  came  from  the  general  officer  level  in  AETC  for  all  instructor  pilots  of  the  courses  being 
studied  to  take  the  on-line  survey.  This  kind  of  high-level  interest  and  support  was  critical  to  achieve  the 
kind  of  response  rates  we  did  and  insure  a  valid  data  set.  The  Figure  1  shows  the  responses  and  estimated 
IP  population  sizes  for  each  of  the  aircraft  included  in  the  study. 
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ORGANIZATION 


Figure  1.  Survey  Response  Numbers 


□  Est.  Population 

□  Responses 


4.0  Quality  Function  Deployment 

In  order  to  assess  the  training  technologies  that  could  best  be  used  in  AETC,  the  researchers  conducted  a 
Quality  Function  Deployment  (QFD)  workshop  with  current  IPs  from  each  of  the  training  aircraft  being 
studied.  When  possible,  the  same  IPs  from  the  survey  construction  workshop  were  brought  back  for  the 
QFD.  Table  5  shows  the  experience  of  the  IPs  who  participated  in  the  QFD  workshop. 


Table  5.  QFD  Workshop  Participants 


Aircraft 

Training  Course 

Base 

Primary 

Aircraft 

Military  Flight  Hours 

T-6 

SUPT 

Moody  AFB  (USN  pilot) 

SH-60B 

1500 

SUPT 

Moody  AFB 

B-52 

4150 

T-l 

SUPT 

Vance  AFB 

KC-135 

2500 

SUPT 

Laughlin  AFB 

C-130 

2000 

SUPT 

Laughlin  AFB 

KC-135 

2500 

T-38A 

SUPT 

Columbus  AFB 

FAIP 

600 

SUPT 

Vance  AFB 

F-16 

1100 

T-38C 

IFF  (Pilot  and  WSO) 

Moody  AFB 

F-15E 

1200 

F-15 

B-Course 

Tyndall 

F-15 

1800 

B-Course 

Tyndall 

F-15 

1800 

F-16 

B-Course 

Luke  AFB 

F-16 

3400 
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B -Course 

Mesa  Research  Site 
(Reservist;  former  IP) 

F-16 

2800 

B-Course 

Springfield  ANG 

F-16 

2500 

FAC-A,  MANTIRN,  NVG 

Luke  AFB 

F-16 

2600 

Average  Flight  Hours: 

2175 

The  purpose  of  the  QFD  was  to  have  selected  IPs  score  the  usefulness  of  various  simulation  technologies 
to  support  the  training  of  each  task  included  in  the  survey  of  their  aircraft.  The  27  technologies  were 
identified  and  defined  by  engineers  and  scientists  from  AFRL.  They  selected  different  simulation 
technologies  that  have  the  potential  for  being  used  for  training  various  flying  tasks.  The  findings  from  the 
simulator  field  evaluations  helped  the  researchers  define  the  candidate  list  of  technologies.  Some  of  the 
technologies  already  exist  and  are  used  in  the  field  by  only  some  of  the  simulators;  others  are  newer 
technologies  available  today  but  are  not  being  exploited  yet;  and  a  few  are  technologies  that  are  not  quite 
mature  enough  but  shoidd  be  in  2-3  years.  To  begin  the  QFD  workshop,  AFRL  provided  definitions, 
demonstrations,  and  discussions  of  the  27  different  technologies  to  ensure  that  all  the  IPs  had  a  common 
and  thorough  understanding  of  the  technologies  and  their  capabilities.  The  27  different  technologies 
scored  in  the  QFD  were  as  follows: 

1.  High  Physical  and  Functional  Fidelity  Cockpit 

2.  Networked  Simulators 

3.  Simulators  Networked  to  Other  Virtual  Assets 

4.  Motion  Platform 

5.  Force  Cueing  Devices 

6.  Cockpit  Environmental  Sounds 

7.  PC-Based  Part-Task  Trainer 

8.  PC-Based  Part-Task  Trainer,  with  Voice  Interaction 

9.  PC-Based  Part-Task  Trainer,  with  Intelligent  Tutoring 

10.  Visual  Systems  with  20/20  Visual  Acuity 

11.  72°  HorizontaP62°  Vertical  Field-of-View  Visual  Display 

12.  220°  (Horiz)  FOV  Visual  Display 

13.  360°  (Horiz)  FOV  (or  as  limited  by  aircraft  design)  Visual  Display 

14.  Head-Mounted  Display  (a.k.a.  Virtual  Reality) 

15.  Eye -Tracking  Technology 

16.  Photo-Realistic  Visual  Database 

17.  Accurate  Sensor  Simulations 

18.  Realistic  Weather  and  Atmospheric  Effects 

19.  3-D  Ground  Models 

20.  Computer  Generated  Moving  Models 

2 1 .  Computer  Generated  Interactive  Entities 

22.  Computer  Generated  Adversaries 

23.  High  Fidelity  Threat  and  Electronic  Combat  Environment 

24.  Mission  Planning  System  Integration 

25.  Digital  Debriefing  System 

26.  Virtual  Reality  Brief/Debrief  Tool 

27.  Current  Part-Task  Trainers 

To  conduct  the  QFD,  the  IPs  broke  into  small  groups  based  on  their  aircraft  types  with  AFRL  facilitators 
in  each  group.  IFF,  F-15,  and  F-16  IPs  were  in  the  same  workshop  group  because  the  majority  of  their 
training  tasks  are  the  same  across  platforms.  The  IPs  were  instructed  to  score  the  tasks  individually  and 
then  discuss  their  ratings  within  the  group  to  then  arrive  at  a  consensus  score.  Forcing  the  groups  to 
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arrive  at  a  consensus  score,  rather  than  just  calculating  an  average  score,  drove  a  great  deal  of  discussion 
and  exploration  amongst  the  participants.  This  discussion  brought  out  different  ideas  on  technology 
application  that  might  not  have  been  considered  or  adequately  factored  into  the  scores  had  the  groups 
been  allowed  to  use  a  more  expedient  averaging  process. 

IPs  were  instructed  to  score  each  technology  based  on  the  following  scale  of  usefulness: 

1  -  Not  at  all  useful 
2- 

3  -  Somewhat  useful 
4- 

5  -  Very  useful 

They  were  also  instructed  to  score  the  technology  independent  of  cost  and  to  ignore  whether  they  already 
had  the  technology  at  their  base  or  not.  The  raw  QFD  scores  should  just  reflect  how  useful  a  technology 
would  be  for  training  a  given  task.  A  cost-benefit  analysis  is  not  a  part  of  this  analysis  model.  It  is  up  to 
the  modernization  planners  and  system  acquisition  personnel  to  work  with  the  contractors  to  determine 
estimated  costs  and  assess  the  benefits.  The  results  of  this  analysis  show  the  planners  how  that 
technology  can  best  be  used  for  the  greatest  gains. 

4.1  QFD  Results  and  Technology  Prioritization 

The  technology  usefulness  scores  arrived  at  by  consensus  for  each  task  during  the  QFD  were  then 
multiplied  by  a  priority  weighting  factor.  The  priority  weighting  factor  was  derived  from  the  data 
collected  with  the  task  survey  described  above  that  was  given  to  all  IPs  across  AETC.  The  priority  score 
was  created  by: 

•  finding  the  mean  IP  rating  for  each  task  on  each  question  (e.g.,  all  of  the  IPs  rated  Task  X  on 
Question  1;  the  mean  rating  of  Task  X  on  Question  1  was  calculated)** 

•  adding  the  mean  ratings  of  each  task  from  questions  1,  2,  4,  and  5.  Question  3,  which  asked  IPs 
to  rate  how  often  a  task  contributed  to  busted  rides,  was  not  included  in  the  calculation  of  the 
priority  index  because  the  AFRL  team  thought  that  the  question  was  too  speculative  in  nature  and 
their  was  little  coherence  in  the  data  collected. 


For  example,  if  4  IPs  were  asked  to  answer  questions  1,  2,  3,  and  4  (Ql,  Q2,  Q3,  and  Q4)  about  tasks  1,  2, 
and  3,  the  resulting  raw  data  would  look  like  this: 


TASK  1 

TASK  2 

TASK  3 

Ql 

Q2 

Q3 

Q4 

Ql 

Q2 

Q3 

Q4 

Ql 

Q2 

Q3 

Q4 

IP1 

2 

2 

3 

2 

2 

1 

1 

3 

2 

3 

2 

3 

IP2 

2 

3 

4 

3 

3 

2 

2 

3 

4 

4 

3 

4 

IP3 

3 

4 

1 

4 

4 

3 

3 

3 

3 

2 

4 

5 

IP4 

3 

3 

1 

3 

3 

2 

2 

1 

3 

1 

1 

4 

Next,  the  mean  rating  for  each  question  on  each  task  is  calculated: 
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TASK  1 

TASK  2 

TASK  3 

Ql 

Q2 

Q3 

Q4 

Ql 

Q2 

Q3 

Q4 

Ql 

Q2 

Q3 

Q4 

AVG 

2.5 

3 

2.5 

3 

3 

2 

2 

2.5 

3 

2.5 

2.5 

4 

**NOTE:  Before  the  means  were  derived,  the  ratings  collected  in  the  survey  process  were  rescaled  so  that  each  one 
was  directionally  congruent;  on  some  of  the  rating  scales,  a  “5”  was  a  positive  attribute  and  on  other  questions  a  “5” 
was  a  negative  attribute.  The  ratings  were  rescaled  so  that  higher  ratings  indicated  more  problematic  tasks. 


Then,  the  mean  ratings  for  each  question  are  added  together  to  give  a  priority  weighting  factor: 


TASK  1 

TASK  2 

TASK  3 

2. 5+3+2. 5+3= 

3+2+2+2.5= 

3+2. 5+2. 5+4= 

PRIORITY 

WEIGHTING 

11 

9.5 

12 

The  higher  the  priority  weight,  the  more  problematic  a  task  is  and  the  greater  the  target  of  opportunity  for 
training  enhancement  with  advanced  simulation  technology.  In  our  example  above,  Task  3  would  be  the 
greatest  target  of  opportunity  for  training  enhancement  with  advanced  simulation  technology. 

Each  of  the  raw  technology  usefulness  scores  assigned  by  IPs  during  the  QFD  were  then  multiplied  by  the 
priority  weighting  factor  for  each  task,  thereby  creating  weighted  priority  scores  that  are  composed  of  a 
technology  usefulness  score  (from  the  QFD)  and  task  ratings  (from  the  IP  task  survey).  These  weighted 
priority  scores  for  each  technology  were  then  added  to  yield  a  total  weighted  priority  score  for  each 
technology.  A  higher  score  indicates  that  a  technology  has  a  greater  potential  for  positive  training  impact 
for  that  course. 

Continuing  our  example,  if  3  training  technologies  (TECHNOFOGY  1,  TECHNOFOGY  2,  and 
TECHNOFOGY  3)  were  assigned  usefulness  scores  during  a  QFD,  the  technology  usefulness  scores  for 
each  task  would  be  multiplied  by  the  priority  weighting  factor  for  each  task;  the  resulting  weighted 
priority  scores  for  each  technology  would  then  be  summed  to  give  a  total  weighted  priority  score  for  each 
technology: 


TECHNOLOGY  1 

TECHNOLOGY  2 

TECHNOLOGY  3 

Priority 

Weight 

Factor 

Technology 
Usefulness 
Score  (from 
QFD 

workshop) 

Weighted 
Priority 
Score 
(priority 
score  X 
technology 
usefulness 
score) 

Technology 

Usefulness 

Score 

Weighted 

Priority 

Score 

Technology 

Usefulness 

Score 

Weighted 

Priority 

Score 

TASK  1 

11 

2 

11*2  =  22 

4 

44 

1 

11 

TASK  2 

9.5 

3 

9.5*3  =  28.5 

4 

38 

3 

28.5 

TASK  3 

12 

4 

12*4  =  48 

5 

60 

4 

48 

TOTAL 

WEIGHTED 

PRIORITY 

SCORE 

22+28.5+ 

48= 

98.5 

142 

87.5 
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In  the  example,  Technology  2  looks  to  have  the  greatest  potential  positive  impact  on  training. 

The  technologies  were  then  ordered  from  highest  to  lowest  total  scores.  This  rank  order  can  help  to 
identify  the  technologies  that  may  have  the  greatest  impact  for  improving  pilot  training.  A  cautionary 
note:  the  resultant  technology  priority  lists  are  only  ordinal  in  nature.  That  is,  technologies  with  higher 
total  weighted  priority  scores  have  a  greater  potential  impact  on  training.  However,  a  technology  with  a 
total  weighted  priority  score  of  400  does  NOT  have  twice  as  much  potential  impact  on  training  as  a 
technology  with  a  total  weighted  priority  score  of  200.  In  addition,  total  weighted  priority  scores  should 
not  be  compared  across  platforms.  Task  surveys  for  different  planes  contained  different  numbers  of  tasks; 
and  since  the  total  weighted  priority  scores  are  products  of  the  number  of  tasks  on  a  survey,  planes  with 
surveys  containing  greater  numbers  of  tasks  would  likely  have  overall  higher  total  weighted  priority 
scores  for  technologies. 

5.0  Conclusion 

No  other  time  in  history  have  we  seen  as  much  readily  available  technology  for  training  warfighters  as  we 
see  today.  The  analytic  process  described  in  this  paper  provides  a  useful  way  of  identifying  those  training 
technologies  with  the  greatest  opportunity  for  enhancing  flying  training  programs.  Critical  to  this  entire 
process  was  involvement  by  the  instructor  pilots  themselves.  Not  only  did  the  survey  responses  from 
over  700  instructors  form  the  basis  for  the  priority  weighting,  the  instructor  participation  in  the  survey 
development  and  technology  scoring  workshops  was  critical.  This  process  has  yielded  a  great  amount  of 
useful  data  and  recommendations  for  the  leadership  in  the  USAF  Air  Education  and  Training  Command. 
The  involvement  by  current  instructors  in  the  workshop  as  well  as  the  analytic  basis  from  instructor 
surveys  yields  great  validity  and  credibility  to  the  analysis  and  recommendations.  The  information  from 
this  process  will  form  the  basis  for  their  simulation  roadmap  and  guide  their  technology  investments  to 
enhance  the  training  of  U.S.  Air  Force  pilots. 


RTO-MP-HFM-101 :  Paper  15 


Page  12  of  12 


